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ABSTRACT 

Disorientation problems influence student learning. To address this issue, this study uses an integrative approach 
to investigate the causes and consequences of disorientation problems so that a complete understanding can be 
obtained. Unlike previous empirical studies, which use statistical techniques, this study attempts to expose 
unexpected relationships with a data mining approach. The results indicate that the causes of disorientation 
problems are associated with learners’ computer experience, the frequencies and time of using each navigation 
tool. On the other hand, the consequences of disorientation problems are concerned with learning effort and 
learning performance. Based on the findings, a framework is proposed to provide guidelines for the development 
of Web-based learning systems that are able to reduce learners’ disorientation problems. 

Keywords: disorientation, data mining, learning performance, computer experience, navigation behavior. 

1. INTRODUCTION 

The Internet is presently being used as an information source for educational purposes (Tutkun, 2011). Thus, 
Web-based learning systems have become increasingly popular in past several years (Liu, Kantarcioglu and 
Thuraisingham, 2008). The major advantage of Web-based learning systems lies within their flexibility. Due to 
this flexibility, many learning platforms have been moved to the Web. This is possibly because Web-based 
learning systems utilize hypermedia structure, which provides a nonlinear way to help each individual develop 
his/her own paths (Deursen and Dijk, 2010). 

However, the nonlinear way may increase complexity so learners may feel confused to interact with the 
Web-based learning systems. One of the major reasons is that the nonlinear ways provide learners with multiple 
directions to reach their learning objectives. Nevertheless, not all of learners can identify a right direction so they 
may feel difficult to choose a right direction and experience disorientation problems (Amadieu et al., 2010). 

Disorientation problems negatively affect student learning in various ways. One of the negative effects is that 
learners cannot identify where they have been, where they are, and where they can go (Zhang and Wang, 2010). 
In other words, learners may feel uncomfortable finding right directions for themselves (Saade and Otrakji, 
2007). In addition, disorientation problems may hinder learners in reaching their desired learning objectives. 
Thus, good performance cannot be achieved which may, in turn, cause learners to lose their motivation and 
confidence while using Web-based learning systems. In the end, learners may also reject the use of Web-based 
learning systems in the future because of these losses (Demirbilek, 2009). 

Due to the importance of these problems, research into this issue has mushroomed lately. A number of studies 
have investigated why learners encounter disorientation problems. For example, some studies claimed that 


Copyright © The Turkish Online Journal of Educational Technology 


1 



TOJET: The Turkish Online Journal of Educational Technology - July 2012, volume 11 Issue 3 



human factors, such as prior knowledge (Mishra and Yadav, 2006) and gender differences (Van Oostendorp and 
Juvina, 2007), may greatly influence the levels of learners’ disorientation problems. Moreover, their navigation 
behavior may be yet another influential factor (Otter and Johnson, 2000). Other studies have indicated that 
learners’ disorientation problems may also have great effects on their learning performance (Ahuja and Webster, 
2001). For instance, Zhang and Wang (2010) indicated that learners who are confused while using such systems 
cannot achieve learning objectives effectively and demonstrate low performance. 

In summary, learners’ disorientation problems are a complex topic related to multiple issues. Therefore, there is 
a need to conduct further investigation to clarify these issues. To this end, the study presented in this paper aims 
to provide a complete understanding of learners’ disorientation problems. To effectively reach our aim, a data 
mining approach is adopted in this study. This is because data mining can discover hidden relationships (Hand et 
al., 2001). In brief, this study employs a data mining approach to investigate why students have disorientation 
problems and how disorientation problems affect their learning. 

This paper is organized as follows. In Section 2, theoretical background is described. Section 3 then moves to 
describe methodology used to achieve our aims. Subsequently, the results related to the causes and consequences 
of disorientation problems are presented in Section 4. The paper then proceeds to Section 5, which proposes a 
framework based on our results. Finally, conclusions are drawn and future work is identified in Section 6. 

2. THEORETICAL BACKGROUND 
2.1. Disorientation Problems 

Previous studies indicate that learners may get lost or become disorientated when using the Web-based learning 
systems (Nielsen and Tahir, 2002). To face this problem, recently, a number of studies investigate why learners 
experience disorientation problems. Their findings include navigation behavior, prior knowledge and gender 
differences. With respect to navigation behavior, Otter and Johnson (2000) found that learners’ disorientation 
problems might become severe when they feel confused with which links they should choose and which 
navigation tools can help them locate relevant information. Moreover, an earlier study by Smith (1996) indicated 
that learners felt confused because the links they selected could not help them locate relevant information. 
Moreover, the search tool they used could not help them find information they want. These findings echoes a 
claim made by Liang and Sedig (2009), which indicated that the investigation of disorientation problems should 
take into account the design of navigation tools. 

With respect to prior knowledge, Mishra and Yadav (2006) found that different levels of learners’ prior 
knowledge can cause different levels of disorientation problems. As shown in their work, learners with the low 
level of prior knowledge can frequently experience disorientation problems while those with the high level of 
prior knowledge can easily avoid disorientation problems. An earlier study by Last et al. (2001) found that 
students with high prior knowledge were better able to navigate easily, remember where they had been, and 
decide how to get to where they wanted to go. Conversely, the students with low prior knowledge often suffered 
from disorientation, not knowing where they had been, or where they could go to find the information that they 
needed. With respect to gender differences, previous research shows that females and males experience different 
levels of disorientation problems. Ford and Miller (1996) found that males were less disoriented than females 
and females showed fewer understandings of how to use navigation tools. The other study by Ford, Miller and 
Moss (2001) indicated that females had difficulties in finding their ways effectively around the Internet and they 
were more likely to get lost and did not feel in control. 

The aforementioned studies indicated that several issues cause the disorientation problems. In addition, recent 
research also examined the consequences of disorientation problems. Some studies investigated the impacts of 
disorientation problems on learning performance. For instance, Amadieu et al. (2010) found that learners’ 
disorientation problems may influence their learning performance. More specifically, learners with serious 
disorientation problems demonstrate low performance while those without disorientation problems show high 
performance. This may be due to the fact that the former can easily lose the path of the subject content while the 
latter can freely locate information to follow the path of the subject contents. Likewise, the other study proposed 
by Ahuja and Webster (2001) indicated that learners with disorientation problems demonstrated low 
performance because they felt difficult to reflect what they had learnt and to locate relevant information they 
wanted. To this end, they failed to integrate the concepts they learned so it was difficult for them to improve 
their learning performance. 

The aforesaid studies explored the causes and consequences of learners’ disorientation problems independently. 
In other words, there is a lack of integrative studies that do comprehensive investigation to examine the causes 
and consequences of disorientation problems together. To this end, this study addresses this issue. Such 
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investigation includes multiple aspects. Thus, there is a need to use intelligent techniques, such as data mining, 
to conduct data analyses so that a global understanding of disorientation problems can be obtained. 

2.2. Data Mining 

Data mining, also known as knowledge discovery (Fayyad and Uthurusamy, 1996), is an interdisciplinary area 
that encompasses techniques from a number of fields, including information technology, statistical analyses, and 
mathematic science (Bohen et al. 2003). A major function of data mining is the search for valuable information 
within large volumes of data (Hand et al., 2001). It can then be used to predict, model or identify 
interrelationships within a set of data (Urtubia et al., 2007) without a need to predefine underlying relationships 
between dependent and independent variables (Chang and Chen, 2005) as some of traditional statistical methods 
require. 

As opposed to the traditional statistical methods, data mining uses the data itself to uncover relationships and 
patterns. In doing so, hidden relationships, patterns, and interdependencies can be discovered, predictive rules 
can be generated, and interesting hypotheses can be found. These are the advantages of data mining (Hedberg, 
1995; Gargano and Ragged, 1999). Due to such advantages, Data mining has been successfully applied in 
various fields, such as image retrieval (Park, Seo, and Jang, 2005), personalized applications (Chang, Chen, 
Chiu, and Chen, 2009), and electronic commerce (Liao, Ho, and Yang, 2009). 

These fields use a variety of data mining techniques, but they may be divided into two major categories: 
supervized learning and unsupervized learning. The former, which is also known as classification, refers to 
assigning objects to predefined categories or classes (Hastie, Tibshirani and Friedman, 2001). On the contrary, 
the latter, which is also known as clustering, is concerned with the division of data into groups of similar objects 
(Chen and Liu, 2008; Nolan, 2002). 

Classification refers to the data mining problem of attempting to discover predictive patterns where a predicted 
attribute is nominal or categorical. The predicted attribute is called the class. Subsequently, a data item is 
assigned to one of a predefined set of classes by examining its attributes (Changchien and Lu, 2001). In other 
words, the objective of classification is not to explore the data to discover interesting segments, but rather to 
decide how new items should be classified. For example, Esposito, Licchelli, and Semeraro (2004) built student 
models for an e-leaming system based on the students’ level of performance: good, sufficient or insufficient. 

Clustering is concerned with the division of data into groups of similar objects. Each group, called a cluster, 
consists of objects that are similar between themselves and dissimilar to objects of other groups. For example, a 
study by Kim (2007) adopted a clustering algorithm to extract sequences with similar behavioral patterns. Their 
results indicated that learners’ navigation patterns are distinctive among different clusters. It implies that 
clustering can provide structural properties to identify dissimilarities between each group. 

Both clustering and classification are useful techniques but a problem of using classification is that there is a 
need to have a predicted attributes. If the predicted attribute was not properly selected, the accuracy of the results 
might be affected. Therefore, we choose to use clustering in our study because disorientation problems are 
affected by multiple aspects and also have multiple effects. 

3. METHODOLOGY DESIGN 

To investigate the causes and consequences of learners’ disorientation problems, an empirical study was 
conducted at a university in Taiwan. This section describes the methodology design of the empirical study, 
including participants, the design of a Web-based learning system, procedures and data analyses. 

3.1. Participants 

50 students voluntarily took part in our study. To recruit these participants, a request was issued to students in 
lectures and further by email, making clear the nature of the study and their participation. Participants had 
diverse levels of Internet skills, background knowledge and computer experience so that we can identify the 
causes of disorientation problems from the perspective of human factors. 

3.2. Web-based learning system 

To be able to measure students’ attitudes to web-based learning, they should have experience in using it (Usta, 
2011). Thus, we developed a Web-based learning system, which introduces the principles of “Interaction design” 
and includes eight sections. The user interface consists of (a) a title bar used to present the subject of the 
Web-based learning system, (b) a navigation-tool panel used to help learners locate information from the 
Web-based learning system, and (c) the main body of the Web-based learning system, which provides the details 
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of the subject content. Figure 1 shows the layout of this Web-based learning system. 


Interaction Design 



3*MkWl 2. 


Nielsen's Heuristics 


Session 2.2.1. HI: Visibility of System Status 

Session 2.2.2. HZ\ Watch between System and the Real World 

Session 2.2.3. H3: User Control and Freedom 

Session 2.2.4. H4: Consistency nnd Standards 

Session 2.2.5. H5: Error Prevention 

Session Z.Z.t. H6: Recognition rather thou Recall 

Session 2.2.7. H7: Flexibility ond Efficiency of Use 

Session 2,2.8. Aesthetic ond Minimalist Design 

Session 2.2.9. H9\ Help users Recognize, Diagnose, end Recover from Errors 

Session Z.Z.1Q. H1Q: Help and Documentation 


Figure 1. The Web-based learning system. 

The Web-based learning system provides multiple navigation tools, which include a main menu, keyword 
search, hierarchical map and alphabetical index. The keyword search is to allow learners to locate information 
for particular concepts with a search box. The hierarchical map uses a graphic to represent relationships among 
various topics. The alphabetic index is to list all of topics in an alphabetic order (Khalifa and Kwok, 1999; 
Nilsson and Mayer, 2002; Chen and Macredie, 2004). In other words, these navigation tools serve different 
purposes so learners are allowed to have freedom to develop their own navigation strategies. By doing so, the 
relationships between learners’ disorientation problems and their navigation strategies can be discovered. 

3.3. Procedure 

The procedure consists of five steps (Figure 2). The details of each step are shown below. 


Filling out personal information 


Taking the Pre-test 


Interacting with the Web-based 
learning system 


Taking the Post-test 


Taking the Questionnaire 



Figure 2. Experimental procedure. 


(1) All of the participants were required to fill out their personal information, e.g., the levels of participants’ 
subject knowledge and system experience. Such personal information was stored in a log file so that the 
relationships between human factors and disorientation problems can be analyzed. 

(2) The participants were required to take a pre-test, which aims to objectively assess their prior knowledge 
of the subject content. The pre-test included 20 multiple-choice questions, each of which includes four 
possible responses: three different answers and an “I don’t know” option. 

(3) The participants were required to interact with the Web-based learning system. During this period, their 
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interaction with the Web-based learning system was recorded in a log file, including their choices of 
navigation tools and the time spent for the Web-based learning system. At the same time, the participants 
were also required to complete practical tasks, which could maintain learners’ motivation (Scanlon, 
2000). Furthermore, performing these tasks is to offer the opportunities of experiencing interface features 
provided by the Web-based learning system. Thus, we were able to identify how learners perceived the 
Web-based learning system. 

(4) After finishing interacting with the Web-based learning system, the participants were required to take the 
post-test to identify how much they had learnt from the Web-based learning system. The post-test was as 
same as the pre-test, which includes 20 multiple-choice questions, each with three different answers and 
an “I don’t know” option. The questions were matched on the pre-test and post-test so that each question 
on the pre-test had a corresponding similar (but not the same) question on the post-test. Creating similar 
questions was achieved by re-writing the question 

(5) A questionnaire has the potential to collect cognitive and affective data quickly and easily (Kinshuk, 
1996). At last, the participants, thus, needed to fill out a questionnaire to identify disorientation problems 
that learners experienced. The questionnaire was developed by an analysis of previous studies on 
disorientation problems (e.g., Saade and Otrakji, 2007; Ahuja and Webster, 2001). There were 16 
questions and each one used a five-point Likert scale ranging very much, quite a lot, average, not much, 
and not at all. 

3.4. Data Analysis 

The aim of this study is to investigate the causes and consequences of learners’ disorientation problems. In order 
to reach our aim, learners’ responses obtained from the questionnaire are worked as attributes to create clusters. 
Among a variety of clustering techniques, K-means algorithm is widely used to partition the data into several 
clusters according to their similarities (Han and Kamber, 2001). In particular, it is frequently applied to analyze 
Web usage data. For example, a study by Wiwattanacharoenchai and Srivihok (2003) used K-means to create 
customer clusters from Web logs of various Internet banking Web sites. Their results showed that there was a 
clear distinction between the clusters, in terms of customer behavior. 

The major principle of the K-means algorithm uses k initial centers, each of which is assigned to each cluster, to 
partition data into k clusters. Each pattern in the cluster is decided based on the nearest distance between the 
pattern and each cluster center. However, a major limitation of using the K-means algorithm is that the number 
of clusters needs to be predefined. In other words, there is a need to identify the most suitable number of clusters 
to perform the K-means algorithm. Such an issue can be treated as parameter exploration (De Jong, 1975), which 
is used to decide the suitable value of parameters. The parameter exploration is useful when a dataset is not 
large. Thus, the K-means algorithm is suitable for this study because the dataset was not large. Therefore, the 
parameter exploration was applied to decide the parameters of the K-means algorithm in this study. 
Subsequently, the number of clusters is set for the large range of value to investigate the robustness of the 
clustering results. The suitable number of clusters is determined based on not only the smallest distance between 
the features in a same cluster, but also the largest distance between the features in different clusters. After doing 
so, we found that the K-means algorithm produces more efficient outcomes for three clusters. 

Subsequently, we select features to examine the corresponding characteristics of each cluster based on a 
comprehensive review by Chen and Macredie (2010). The details of these features are described in Table 1. The 
detailed differences among the three clusters were illustrated with the mean and standard deviation of each 
feature. In addition, the correlation analysis is also conducted to provide additional evidence as to the 
relationships among computer experience, disorientation problem, and learning performance. 


Table 1. The features used to examine each cluster. 


Features 

Explanation 

Human Factors 

• Computer experience 

• Gender Differences 

Navigation Behavior 

• Frequencies of using navigation tools: the total number of times each 
navigational tool used 

• Time of using each navigation tool: the total amount of time each 
navigational tool used 

Learning Effort 

• Total Time spent for reading pages in the Web-based learning system 
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Learning performance 


Post test score: the score obtained from the post-test 

Gain Score: the difference between the post-test score and pre-test score 


4. RESULTS AND DISCUSSION 

Three clusters were created with a K-means algorithm. As shown in Table 2, the percentage of learners within 
each cluster is satisfactory because the number of the members in each cluster is reasonably balanced. The mean 
and standard deviation (STD) of the responses to the questionnaire for each cluster are shown in Table 2. The 
high mean value represents the high level of disorientation problems that learners experience and vice versa. The 
tendency of each cluster is shown below. 


^able^ScorcJromquestionnaireJorEachClu^ 


Cluster (C) 

Mean 

STD 

Instances 

Cl 

2.33 

0.46 

21(42%) 

C2 

3.35 

0.36 

17(34%) 

C3 

5.23 

0.47 

12(24%) 


• Cluster 1 (Cl, N=21): Learners do not experience any difficulties while using the Web-based learning 
system. In other words, the learners in Cluster 1 clearly understand the meanings of buttons and icons. 
However, it would be helpful for them if a suggested route could be given through the Web-based learning 
system. 

• Cluster 2 (C2, N=17): Learners are somewhat confused while using the Web-based learning system. 
Because the uses of the back/forward buttons are unclear to them. Consequently, it is difficult for these 
learners to keep track of topics which have been learnt. 

• Cluster 3 (C3, N=12): Learners may be easily confused while using the Web-based learning system. The 
learners in Cluster 3 do not fully understand the meaning of buttons and icons and it is difficult to keep 
track of topics which they have learnt. Moreover, they also think that too many options are provided so 
they do not know how to choose a proper navigation tool to help them locate information. 

The aforementioned results indicate that learners encounter different levels of disorientation problems. Some 
may feel comfortable with the Web-based learning system while others may feel very confused with the system. 
Thus, there is a need to further develop a complete understanding for this issue. To this end, it is necessary to 
investigate why they have disorientation problems and how disorientation problems influence their learning. In 
other words, we need to identify the causes and consequences of learners’ disorientation problems, which are 
presented in subsections below. 

4.1. Causes of disorientation problems 

The results indicate that disorientation problems are caused by the levels of computer experience, the use of 
navigation tools, and time spent reading pages. The details are shown below. 

4.1.1 Computer experience 

The levels of learners’ computer experience in each cluster are illustrated in Figure 3, which shows there is an 
inverse relationship between learners’ computer experience and their disorientation problems (r=-.275; p<.05). 
More specifically, the learners with a high level of computer experience may never feel confused while the 
learners with a low level of computer experience may easily feel confused. A possible reason is that learners who 
have sufficient computer experience can easily understand each function provided by the Web-based learning 
system so they have less confusion. Thus, learners in Cluster 1, who have the higher level of computer 
experience, never feel confused. On the other hand, learners in Cluster 2 and Cluster 3, who have the lower level 
of computer experience, may easily feel confused with using the Web-based Learning system. Our finding 
echoes a claim made by Mishra and Yadav (2006), which indicated that learners’ prior computer experience 
greatly affects the levels of disorientation problems that learners experience. 
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cl c2 c3 


■ with low computer 
experience 

■ with high computer 
experience 


Figure 3. Relationships between learners’ computer experience and their disorientation problems. 


4.1.2. Frequencies of using each navigation tool 

Another influential factor is related to the frequencies of using various navigation tools. Table 3 describes how 
learners in each cluster use each navigation tool. This table shows several interesting tendencies. The first 
tendency is related to the use of the hierarchical map. As showed in this table, learners in Cluster 1 made the 
most use of the hierarchical map while those in Cluster 3 made the fewest use of the hierarchical map. Such a 
difference may be another reason to explain why learners in Cluster 1 experienced fewer disorientation problems 
while those in Cluster 3 experienced more disorientation problems. More specifically, there is an inverse 
relationship between the frequencies of the use of the hierarchical map and the levels of disorientation problems 
that learners experience (i=-.337; p<.01). In other words, learners who frequently use the hierarchical map feel 
less confused than those who rarely use the hierarchical map. These findings are in line with those of the study 
by Amadieu et al., (2010), which found that the hierarchical map is useful to reduce learners’ disorientation 
problems. This is due to the fact that the hierarchical map presents an overall picture, which shows the 
relationship between each topic. Such an approach can help learners find a right direction for their learning paths 
so that learners can avoid disorientation problems. 


Table^Frcquencyofusi^ 


Frequency of each navigation tool used 


Cluster (C) 

Keyword 

Menu 

Index 

Map 

Mean 

STD 

Mean 

STD 

Mean 

STD 

Mean 

STD 

Cl 

2.33 

0.06 

1.62 

0.2 

0.48 

0.11 

0.86 

0.06 

C2 

2.18 

0.33 

1.76 

0.15 

0.18 

0.09 

0.59 

0.8 

C3 

2.42 

0.08 

1.33 

0.89 

0.17 

0.09 

0.33 

0.89 


The other tendency is concerned with the balanced use of each navigation tool. As showed in Figure 4, learners 
in Cluster 1 tend to evenly use each navigation tool while those in Cluster 3 tend to rely on a particular 
navigation tool. Such a difference may be the other reason to explain why learners in Cluster 1 experienced 
fewer disorientation problems while those in Cluster 3 experienced more disorientation problems (Gwizdka and 
Spence, 2007). Each navigation tool shows different ways to help learners construct their knowledge. Thus, 
learners in Cluster 1, who use multiple tools, can organize information in a more effective way. In contrast, 
learners in Cluster 3, who mainly focus on using the keyword search, may have difficulties in developing 
effective learning strategies. In particular, the keyword search can only present a single concept, instead of 
illustrating relationships between each topic so they may fail to see logical relationships between each topic and 
can experience more disorientation problems. 


In summary, the aforementioned tendencies indicate that learners, who tend to use multiple tools to locate 
information, can avoid experiencing disorientation problems while the learners, who tend to rely on a single tool, 
such as the keyword search, can easily encounter disorientation problems. Furthermore, the learners, who 
frequently use the hierarchical map, feel less confused than those who rarely use the hierarchical map. In other 
words, the hierarchical map may be beneficial to reduce learners’ disorientation problems. 
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4.1.3. Time of using each navigation tool 

The other critical factor is concerned with the time spent for each navigation tool. As shown in Figure 5, most of 
the learners tend to spend more time using the keyword search and main menu than the other two tools. Unlike 
other clusters, learners in Cluster 3 tend to spend more time for using the keyword search than the main menu. 
As indicated in the previous section, the keyword search mainly emphasizes on a single topic, instead of giving 
the whole picture of the subject content. Therefore, it is difficult for them to logically connect each topic with the 
keyword search. Moreover, they spend little time for using the main menu, which can help learners construct 
logical relationships between each topic. Such a reason may explain why learners in Cluster 3 may experience 
more disorientation problems than those in Cluster 1. 

In addition to the keyword search and the main menu, the learners in Cluster 3 also spend less time for using the 
hierarchical map than those in other clusters. This may be another reason to explain why learners in Cluster 3 
experienced more disorientation problems than those in other clusters. These results are in agreement with those 
presented in Section 4.1.2, which indicate that the hierarchical map may be beneficial to reduce learners’ 
disorientation problems. 

On the other hand, learners in Cluster 3 never spend time for using the alphabetical index (Table 4). As 
mentioned in the result of Section 4.1.2, the unbalanced use of their navigation tools can cause disorientation 
problems because each navigation tool serves different purposes. 


2.50 

2.00 

1.50 

1.00 

0.50 

0.00 



cl c2 c3 


Figure 5. Times using each navigation tool. 


■ keyword 

■ menu 

■ index 

■ map 
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In summary, this section demonstrates that the level of computer experience, the choices of navigation tools and 
time spent for each navigation tool may affect the levels of disorientation problems that learners experience. 
Although those findings provide a deep understanding of the causes of disorientation problems, there is also a 
need to discover the consequences of disorientation problems, which are presented in the following section. 

Table 4, Time of using each navigation tool. 


Time of each navigation tool used 


Cluster (C) 

Keyword 

Menu 

Index 

Map 

Mean 

STD 

Mean 

STD 

Mean 

STD 

Mean 

STD 

Cl 

2.38 

0.02 

2.1 

0.3 

0.48 

0.98 

1.00 

0.22 

C2 

2.24 

0.31 

1.94 

0.25 

0.53 

0.18 

0.65 

0.17 

C3 

2.17 

0.11 

1.25 

0.14 

0 

0 

0.25 

0.87 


4.2. Consequences of disorientation problems 

The results indicated that disorientation problems can influence the learners’ effectiveness and their 
performance. The details are shown below. 

4.2.1. Learning Effort 

As shown in Table 5, learners in Cluster 3, who have a high level of disorientation problems, spend the least 
time reading pages in the Web-based learning system. Conversely, learners in Cluster 1, who have a low level of 
disorientation problems, spend the most time reading pages in the Web-based learning system. More specifically, 
the learners who meet serious disorientation problems may feel confused with how to use the Web-based 
learning system so it is difficult for them to concentrate on the content. This is the reason why they spent the 
least time for reading pages. In contrast, the learners who do not meet serious disorientation problems can easily 
concentrate on the content so they spent the most time for reading the pages. Such a difference suggests that the 
former put more effort to read pages presented in the Web-based learning system than the latter. 


Table 5. Total time spent for reading pages. 


Cluster (C) 

Mean 

STD 

Cl 

15.95 

1.53 

C2 

15.35 

1.91 

C3 

13.67 

1.12 


4.2.2. Learning Performance 

This Section describes how learners’ disorientation problems affect their learning performance based on their 
post-test scores and gain scores. The former is concerned with their performance in the post-test while the latter 
emphasizes on the improvement that the learners have made. As shown in Table 6, learners in Cluster 3 obtain 
the lowest post-test score while those in Cluster 1 demonstrate the highest post-test score. Likewise, learners in 
Cluster 1 have the highest gain score whereas those in Cluster 3 have the lowest gain score (Table 7). It suggests 
that the results from the post-test are consistent with those from the gain score. 


Table 6. Post-test Score for Each Cluster. 


Cluster (C) 


Post-test Score 


Mean 


STD 


Instances 


Cl 

14.95 



2.77 

21(42%) 

C2 

12.65 



2.64 

17(34%) 

C3 

10.92 



3.26 

12(24%) 

Table 7. Gain score from Performance for Each Cluster. 

Cluster (C) 


Gain Score 


Instances 

Mean 



STD 

Cl 

5.29 



1.24 

21(42%) 

C2 

4.82 



1.83 

17(34%) 

C3 

3.25 



3.57 

12(24%) 
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In summary, learners in Cluster 3 show worse learning performance than those in the other two clusters. As 
mentioned in the above section, learners in Cluster 3 also experienced more disorientation problems than those in 
the other two clusters. It implies that the learners with more disorientation problems do not perform as well as 
those without disorientation problems. The finding echoes a claim made by Last et al., (2001), which indicated 
that disorientation problems have greatly effects on students’ learning performance. In other words, there are 
inverse relationships between disorientation problems and students’ learning performance, regardless the 
post-test score (r=-.337, p<.01) or gain score (r=-.227, p<05). A possible reason is that learners with less 
confusion can properly choose navigation tools to locate relevant information from the Web-based learning 
system. As showed in Section 4.1.2, learners in Cluster 3 rarely use the main menu and hierarchical map, which 
are helpful to build logical relationships between each topic. Thus, they may fail to see the logical relationships 
between each topic, which, in turn, their performance is not as good as those in other clusters. 

In addition to showing the lowest learning performance, the performance of learners in Cluster 3 also reveals 
high variation because the standard deviation of the gain score in Cluster 3 is very high. It implies that 
disorientation problems have great effects on student learning because they cannot only negatively affect 
learning performance but also increase their diversities. 

5. THE DEVELOPMENT OF A FRAMEWORK 

This study contributes to give a deep understanding of the causes and consequences of learners’ disorientation 
problems. According to our results, learners’ computer experience, the frequencies and time of using each 
navigation tool may be the major causes of disorientation problems. On the other hand, disorientation problems 
also have great effects on learning effort and learning performance. To this end, there is a need to remove the 
causes of disorientation problems so that the consequence can also be avoided. The following guidelines are 
proposed to address this issue. 

5.1. To provide annotation for each navigation tool 

As showed in our results, the learners with less computer experience may easily encounter disorientation 
problems. It may be because they lack sufficient computer experience to identify the functions of each 
navigation tool. Thus, they have difficulties to find useful tools for themselves (e.g., Jenkins, Corritore, and 
Wiedenbeck, 2003; McDonald and Stevenson, 1998) and need to spend much time on trying various navigation 
tools. This is the reason why they cannot concentrate on reading pages in the Web-based learning system. To 
address this problem, the Web-based learning system should provide clear description for each navigation tool so 
that learners can know the function of each navigation tool. For example, Annotations, which can work as a label 
to support local orientation by providing additional description, can be used to describe the function of each 
navigation tool (Chen and Macredie, 2002). In particular, the annotations can be used together with the 
hierarchical map, which can help learners easily see relationships between each topic and avoid disorientation 
problems. 

5.2. To provide direct guidance for suitable tools 

The learners, who tend to focus on using particular navigation tools, may easily encounter disorientation 
problems. In particular, only using the keyword search may make the learners obtain the fragment information of 
subject content and fail to build logical relationships between each concept and to get the overall picture of the 
subject content. To address this problem, the Web-based learning system should suggest suitable navigation 
tools for learners. For example, Direct Guidance, which guides the learners to the next “best” item (Brusilovsky, 
1998), can be used to recommend suitable navigation tools. In particular, the main menu and hierarchical map 
should be recommended to learners with disorientation problems because these two navigation tools can 
illustrate the logical relationships between each concept. 

A framework (Figure 6) is proposed based on the above listed two guidelines. 



Figure 6. The proposed framework. 
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6. CONCLUDING REMARKS 

This study aimed to use an integrative way to investigate learners’ disorientation problems. The causes and 
consequences of disorientation problems are examined by using the K-mean clustering method. This study 
demonstrates some interesting findings related to these two issues. Regarding the causes of disorientation, 
learners’ computer experience, the frequencies and time of using each navigation tool greatly affect their 
disorientation problems. Regarding the consequences of disorientation problems, the disorientation problems 
have major impacts on learning effort and learning performance. According to our findings, the framework is 
proposed to support the designers to build a Web-based learning system which can not only reduce the causes of 
learners’ disorientation problems but also avoid their consequences. 

Although this study makes significant contributions, there are still some limitations. Firstly, this is a small-scaled 
study. There is a need to consider a larger sample to provide additional evidence as to the use of navigation tools 
in future research. Moreover, this study only uses K-mean clustering method to discover the causes and 
consequences of learners’ disorientation problems so further works can consider other clustering algorithms, e.g., 
fuzzy C mean (Liu and Xu, 2008). In addition, we can further apply other data mining methods to analyze 
learners’ disorientation problems, e.g., association rules or classification. Such results can be integrated into the 
framework presented in Section 5 so the robustness can be enhanced. 
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